Method for 3D mapping of 2D point of interest

ABSTRACT

A method for mapping 2-dimensional point of interest to a 3-dimensional view. The disclosed method includes capturing a 3-dimensional (3D) image with an augmented reality device. Matching images of a 2D image database with the captured 3D-image, the 2D image database containing 2-dimensional (2D) images associated with points of interest having a 2-dimensional (2D) data set. When the 2D-image matches the 3D image, capturing 3D-data for the matched 2D-image and converting the 2D-data set of the point-of-interest to a 3D-data set. The disclosed method simplifies the configuration of points of interest for 3D applications, such as an augmented reality device.

The present invention relates to a method for 3D mapping of 2D Point ofinterest and an augmented reality device. In particular, the methodrelates to mapping a point of interest defined for a 2-Dimensional imagein a 3-Dimensional image captured by an augmented reality device.

BACKGROUND

In mapping and navigation applications, points of interest are commonlyapplied to indicate where a particular object may be found. Such objectsmay be resources, such as gas stations, restaurants, or may be parts orcomponents. A rough indication of the location where such objects are tobe found is usually enough, but in certain conditions this may not besufficient. In particular, in factory sites or other type ofmanufacturing plants, the precise location of components present in amachine, control cabinet or other industrial equipment, is required toprevent unintentional damage or even hazardous situations. The knowledgeof where precisely to find and recognize parts in such circumstances, isof high importance. Prior art solutions include layout drawings andmachine diagrams, pointing out the layout or structure of a particularmachine. Enhanced versions thereof include actual pictures i.e. photoimages of the particular machine or electrical control box. However, aconfiguration may be altered over time. For example, when parts arereplaced or components are updated.

SUMMARY OF INVENTION

To address the above mentioned needs, operator and maintenance personnelmay be instructed to re-take an image each time a particular machine,control cabinet or equipment is inspected or at least interacted with.Or, when new equipment is installed or equipment is updated, to takepictures of the newly installed configuration.

With these images Points of interest may be associated by assigningcoordinates within the captured images. The operator may identifypoints-of-interest in the image at the moment of taking a picture, or itmay also be performed by at a later stage. Or it may be performed byimage-recognition software capable of identifying predefined components,equipment or other relevant parts that are of interest. Either way,these points of interest are defined in a 2D image that relate to acertain relevant part or piece of information. This allows creating adatabase of images having associated Points of interest that are definedin a 2-dimensional (2D) space. Hence, the captured image is treated as a2D canvas i.e. surface space which may spanned by Cartesian coordinatesor defined by any other suitable coordinate system.

It is an object of the invention to further alleviate the definition ofpoints of interest for 3-dimensional applications.

According to one aspect, there is provided a method for mapping2-dimensional point of interest to a 3-dimensional image. The methodincludes capturing a 3-dimensional (3D) with an augmented realitydevice. Matching images of a 2D image database with the 3D, the 2D imagedatabase containing 2-dimensional (2D) images associated with points ofinterest having a 2-dimensional (2D) data set. And when the 2D-imagematches the 3D, capturing 3D-data for the matched 2D-image andconverting the 2D-data set of the point-of-interest to a 3D-data set.

The disclosed method simplifies the configuration of Points of interestfor 3D applications, such as an augmented reality device. As e.g. onlyX,Y coordinates need to be associated with a 2D image.

The invention further reduces the effort of defining Points of interest,as this may still be done in 2D images. Or may have been donepreviously, thus allowing the re-use of a prior created database with 2Dimages containing points of interest.

The invention further reduces the amount of data required to be storedas only 2D images and associated 3D absolute position coordinates are tobe stored, and not full 3D images and associated content.

In this application, any 3D image has a 3D representation consisting ofa 2D image i.e. 2D frame with associated depth information for a numberof points or all points in the 2D frame.

BRIEF DESCRIPTION OF THE DRAWINGS

By way of example only, the embodiments of the present disclosure willbe described with reference to the accompanying drawing, wherein:

FIG. 1 illustrates schematically an example of an augmented realitysystem in an industrial control system;

FIG. 2 illustrates schematically an example of an augmented realitysystem in accordance with the invention;

FIG. 3 illustrates an example of a method in accordance with theinvention;

FIG. 4 illustrates another example of a method in accordance with theinvention;

FIG. 5 illustrates an example of a method for creating a 2D imagedatabase;

FIG. 6 illustrates schematically a matching of a 2-dimesnional image ina 3-dimensional coordinate system; and

FIG. 7 shows a flow diagram illustrating a method in accordance with theinvention.

DETAILED DESCRIPTION

Referring to FIG. 1, various components typically making up anindustrial control system 1 are shown. There are an automation server 2,a SCADA station 3, and cabinets 4, 5 with Programmable Logic Controllersand other equipment. Also shown is an augmented reality image server 6and an augmented reality device 7, which make up an augmented realitysystem in an industrial control environment. An operator may use thedevice 7 to assist when inspecting equipment in the cabinets 4, 5 byplacing the device 7 in front of the object under inspection. The device7 is intended to capture an image of the equipment, recognize theequipment in the image, communicate with the image server 6, and showadditional information regarding the equipment recognized. In thismanner, an augmented reality may be obtained, as any informationrelating to the equipment may be retrieved and shown as an overlay inthe image shown on the display of the device 7.

In FIG. 2, the augmented reality device 7 and augmented reality imageserver 6 of the augmented reality system are shown schematically in moredetail. The augmented reality image server 6 has a 2D database 8containing 2-dimensional images with points of interest having2-dimensional (2D) data sets, and a 3D database 9 containing3-dimensional images with points of interest having 3-dimensional (3D)data sets.

The augmented reality i.e. 3D device 7 has an image sensor 10, anadjustable lens 11, and 3D-sensors 12 which are able to determine adepth i.e. distance between the 3D device and a visible object. The 3Ddevice further has an absolute positioning sensor 13, a GPS sensor inthis example, and a display 14. Further shown in FIG. 2 are a 2D imagerecognition module 15, a 3D data capture module 16, and 2D to 3Dconverter module 17. In this example, these are shown as separatehardware modules being part of the 3D device 7. In another example,these modules may be implemented as software modules executed by ageneral processing unit. In yet another example, these modules may bepart of the augmented reality image server 6 or located on a separateserver as part of an augmented reality system.

In this example the absolute position sensor is a GPS sensor, but othernavigational sensors capable to determine the position of the device inrelation to a reference point are also suited, such as e.g. the EUGalileo positioning system or China's BeiDou Navigation SatelliteSystem. Other position sensors that allow to obtain an absolute positionwithin a geographical space are also suited, such as position sensor andsystem that allows to define a fixed known reference point and determinethe device position relative to that defined reference position.

Referring to FIG. 3, a method for mapping a 2-dimensional point ofinterest in a 3-dimensional view is shown. The method may be implementedby the augmented reality system examples described with reference toFIG. 2. The method starts with capturing 102 a 3-dimensional (3D) imagewith an augmented reality device. Followed by matching 103 images of a2D image database with the captured 3D-image. This 2D image databasecontains 2-dimensional (2D) images associated with points of interesthaving a 2-dimensional (2D) data set. When the 2D-image matches the 3Dimage, the method continues by capturing 3D-data 104 for the matched2D-image and converting 105 the 2D-data set of the point-of-interest toa 3D-data set. The obtained 3D data set will store the points ofinterest with global position coordinates, such as e.g. GPS-coordinates,which will allow easy retrieval of points of interest based on theglobal position of the augmented reality device.

The above steps may be regarded as a learning phase wherein the 3D database on the AR-server is populated with 3D data sets of the points ofinterest available for later retrieval. In this manner, 2D imagerecognition of one image is only required once, as the 3D data set ofeach point of interest will be stored with global position coordinates.

Turning to FIG. 4, an example of a method for displaying a 2-dimensionalpoint of interest in a 3-dimensional view is shown. The method may alsobe implemented by the augmented reality system examples described withreference to FIG. 2. This method includes, similarly to the method asdescribed in reference to FIG. 3, capturing 202 the 3D image, followedby matching 203 images of the 2D image database with the captured3D-image and, when the 2D-image matches the 3D image, continues bycapturing 204 3D-data and converting 205 the 2D-data set of thepoint-of-interest to a 3D-data set. In addition to the above, the methodincludes providing 201 a 2D image database containing 2-dimensional (2D)images associated with points of interest having a 2-dimensional (2D)data set; and displaying 206 the point of interest in the 3D view asaugmented reality. The 3D view may be shown on the display of theaugmented reality device, e.g. when an operator is performing inspectionor maintenance tasks. The 3D view may also be shown on a display of awork station in a remote control room.

One advantage of the above described methods is that this allows pointsof interest to be defined in 2D images in a very simple manner, while inthe learning phase all points of interest may be instantiated in a3D-image for the application by means of the 2D recognition.

Referring to FIG. 5, a method for creating a 2D image database storing2-dimensional (2D) images associated with points of interest is shown.In order to create such a 2D image database a 2D design tool is used toprocess images that were previously taken e.g. by an operator orotherwise provided 301. In the 2D images points of interest are to bedefined which may for example refer to certain switches, circuitbreakers, I/O connections or other components.

Thereto each image is assigned 302 an image identifier ID-im as a uniquemeans for referring to the image. Further properties as length L andwidth W of the image are determined 303, as is a reference point. Thereference point may be one of corners of the image, it may be the centreof the image, or it may be any other reference point suitable to betaken as an origin for a coordinate system, such as the origin of aCartesian system. In the latter case, with the origin on one corner,maximum values for X and Y will be equal to L and W.

With the 2D image being processed, a point of interest having arespective poi-identifier ID-poi may be defined 304 within the image.Along with the defining of the point of the interest and itspoi-identifier, first coordinates of the point of interest in relationto the reference point, i.e., origin, of the image are determined 305;for example, X-poi and Y-poi, for which purpose the 2D image may beregarded as a Cartesian coordinate space R. The above may be repeatedfor multiple points, as one 2D image may show multiple points ofinterest. In this manner, each point of interest has a 2-dimensional(2D) data set that at least contains the coordinates of the poi in the2D image. Such a data set may for example take the form for each P.o.I.as (ID-poi, X-poi, Y-poi, ID-im).

Creating the 2D image database further requires associating 306 eachdefined point of interest with the image identifier. This may be done bycreating a record for each image identifier and storing allpoi-identifiers of the respective points of interest associated with theimage and its image identifier. It may also be done by creating a recordfor each poi-identifier and storing the image identifier associatedtherewith.

All the above is stored 307 in the 2D image database, so the imageidentifier, the length and width of the image, the reference point inthe image, the poi-identifier, the coordinates of the at least one pointof interest, and the association with each defined point of interest.Accordingly, a 2D image database is provided containing 2-dimensional(2D) images associated with points of interest having a 2-dimensional(2D) data set.

Returning to FIGS. 2 and 3, the method will be described in furtherdetail. In order to capture the 3D-image, a 3D representation isobtained of the captured 3D image via 3D-sensors and the image sensor ofthe augmented reality device. This 3D representation includes a 2Dframe, which in this example is a 2D-image captured by the image sensor,consisting of points defined by two dimensional frame coordinates, forexample values for X and Y in a Cartesian coordinate system. Each pointin the 2D frame is associated with a depth, similar to the Cartesiancoordinate Z, which is obtained by the 3D-sensors. The 2D frame couldalso be constructed as a vector field with a vector Z defined for eachpoint (X, Y). The 2D frame does not necessarily need to be continuous,for example it may also be of a discrete form.

For example, the 3D device may use laser to apply a matrix projection todetermine the deformation caused by objects in front of it. Such aprojection will only obtain values for Z i.e. depth for a limited numberof points depending on the mesh size or pitch of the matrix projection.In yet another way, the 2D frame may be construed as a projection of the3D-image along the Z-axis onto the X-Y plane.

The matching of images of the 2D image database with the 3D-imageincludes comparing each 2D image of the 2D image database with the 2Dframe of the 3D representation by scaling and translating the 2D imageover the 2D frame. An example of the required processing movements isshown in a simplified manner in FIG. 6.

When the 2D image matches the 3D image, so in the above example the 2Dframe of the 3D representation, the 3D data for the corresponding matchneeds to be captured. This capturing of 3D data for the matched 2D imageincludes determining frame coordinates of the reference point of the 2Dimage in the 2D frame of the 3D representation. The capturing furtherincludes determining scale information, and obtaining a distance Dbetween the augmented reality device and the 2D image; both of which maybe obtained from the lens of the 3D device. One common way of performingscaling is a homothetic projection, which allows resizing of a shapewhile maintaining its ratio aspects and with minimal deformation.

The frame coordinates, scale information, distance D and imageidentifier of the matched image may be stored in a matched image objectas an intermediate result.

With the 3D-data captured, the 2D-data set of the point-of-interest canbe converted to a 3D-data set. This converting of the 2D-data set to the3D-data set includes applying the scaling information to the firstcoordinates of the point of interest. The converting further includesapplying the frame coordinates of the reference point to the coordinatesof the point of interest to obtain frame coordinates of the point ofinterest. And finally, to obtain a depth coordinate for the point of theinterest, the depth associated with frame coordinates of the point ofinterest and the scaling information is processed to calculate the depthcoordinate. In this manner, a set of 3-dimensional coordinates of thepoint of interest is obtained, defined in the coordinate system of theaugmented reality device R′.

These frame coordinates may be converted to global position coordinates,such as e.g; known as GPS. Thereto, a global position of the augmentedreality device is obtained. From this a global position of the point ofinterest is obtained by applying the global position of the augmentedreality device to the frame and depth coordinates of the point ofinterest. These 3D data may then be stored as the global position of thepoint of interest in the 3D data set.

FIG. 6 shows the relation of the coordinate space R of the 2D image 18and a possible coordinate space R′ of the augmented reality device. Inanother example, instead of Cartesian coordinates, the device coordinatespace could also be defined by spherical coordinates.

Referring to FIG. 7, a flow chart is shown illustrating a furtherexample of a method in accordance with the invention. The 2D design tool19 is used to define point-of-interests for 2D images. The 2D designtool 19 delivers these 2D images to the 2D image database 8 in whichthey are stored, together with an image identifier ID-im, the length L,the width W. In addition, the points-of-interests are stored togetherwith, for each point-of-interest, a poi-identifier ID-poi, the Xcoordinate of the p.o.i. in the 2D image X-poi, the Y coordinate of thep.o.i. in the 2D image Y-poi, and the image identifier ID-im for whichthey were defined and hence are associated. The 3D image is captured 405with X, Y, Z coordinates for each point of the image.

The 2D images are processed one by one by the 2D recognition tool to bematched 401 with captured 3D image. When matched a match-image object iscreated containing the image identifier ID-im, updated coordinates X′and Y′ defining the coordinates of the origin in the captures 3D image,and scale-information defining the scale of the 2D image in the captured3D image. As the captured 3D image may show multiple points-of-interest,the 3D image is compared with further 2D images of the 2D imagedatabase; if further matches are obtained these are also stored in thematch-image object. Therefore, in case of a match it is checked whetherthis is a new match 402 or not. As the location of the 3D device may beknown the number of 2D images relevant to be compared may be limitedbased on the location.

When the 2D image is matched to the 3D image the 2D data X-poi, Y-poiare converted 403 by the 2D to 3D module to one 3D data set which isstored in the 3D database 9. The conversion is performed by processingthe match-image objects for each 2D image associated with the 3D image.The 2D image matching may by itself also contain multiplepoints-of-interest, in which case the converting of the 2D data set isapplied to all points-of-interest defined for that image. All theconverted 3D data sets are stored in the 3D database 9.

When a 2D image was previously matched to 3D image, the converting maybe skipped as prior converted 3D data set will be available. Hence, onlywhen a new matched image is obtained the converting of the 2D to 3D dataset will be performed.

Once the 3D data set is obtained, the 3D image will be shown on thedisplay of the 3D device together with all identifiedpoints-of-interest.

Although the present invention has been described above with referenceto specific embodiments, it is not intended to be limited to thespecific form set forth herein. Rather, the invention is limited only bythe accompanying claims and, other embodiments than the specific aboveare equally possible within the scope of these appended claims.

Furthermore, although exemplary embodiments have been described above insome exemplary combination of components and/or functions, it should beappreciated that, alternative embodiments may be provided by differentcombinations of members and/or functions without departing from thescope of the present disclosure. In addition, it is specificallycontemplated that a particular feature described, either individually oras part of an embodiment, can be combined with other individuallydescribed features, or parts of other embodiments.

The invention claimed is:
 1. A method for mapping a 2-dimensional (2D)point of interest in a 3-dimensional (3D) view, comprising: in alearning phase, capturing 3D images that match 2D images previouslystored in a 2D image database and associating the 3D images with the 2Dimages in the 2D image database; outside of the learning phase:capturing a 3D image with an augmented reality device, the captured 3Dimage comprising a 2D image captured by an image sensor and associateddepth information captured by a laser 3D sensor based on a matrix ofpoints corresponding to the captured 2D image, the captured 2D image andassociated depth information being a 2D frame constructed as a vectorfield with a vector corresponding to depth associated with each point inthe 2D image; matching the 2D images of the 2D image database with thecaptured 3D image based on the 3D images associated with the 2D imagesin the 2D image database, the 2D image database comprising 2D imagesassociated with points of interest having a 2D data set, wherein when a2D image of the 2D image database matches the captured 3D image:capturing 3D data for the matched 2D image, and converting the 2D dataset to a 3D data set, wherein the captured 3D image and the images inthe 2D data base are in spherical coordinates.
 2. The method accordingto claim 1, wherein the 2D image database is created by: providing the2D images; and for each image of the 2D images: assigning an imageidentifier, determining length and width of the image, defining at leastone point of interest having a respective poi-identifier, determiningfirst coordinates of the at least one point of interest in relation to areference point of said each image, associating each defined at leastone point of interest with the image identifier, and storing the imageidentifier, the length and width, the reference point, thepoi-identifier, the first coordinates of the at least one point ofinterest, and the association with said each defined at least one pointof interest.
 3. The method according to claim 1, wherein capturing the3D image comprises obtaining a 3D representation of the captured 3Dimage via 3D sensors of the augmented reality device, wherein the 3Drepresentation comprises a 2D frame consisting of points defined by 2Dframe coordinates, and wherein each point of said points is associatedwith a depth.
 4. The method according to claim 1, wherein the matchingimages of the 2D image database with the captured 3D image comprisescomparing each 2D image of the 2D image database with a 2D frame of the3D representation by scaling and translating said each 2D image over the2D frame.
 5. The method according to claim 1, wherein the capturing 3Ddata for the matched 2D image comprises: determining frame coordinatesof a reference point of the matched 2D image in a 2D frame; determiningscale information from a size of the 2D image and a size of the 2Dframe; and obtaining a distance between the augmented reality device andthe 2D image.
 6. The method according to claim 1, wherein converting the2D data set to the 3D data set comprises: applying scaling informationto first coordinates of the points of interest; and applying framecoordinates of a reference point to the first coordinates of the pointsof interest to obtain frame coordinates of the points of interest. 7.The method according to claim 6, wherein converting the 2D data set tothe 3D data set further comprises determining a depth coordinate for thepoints of interest from: a depth associated with the obtained framecoordinates of the points of interest; and the scaling information. 8.The method according to claim 7, wherein converting the 2D data set tothe 3D data set further comprises: obtaining an absolute position of theaugmented reality device; determining an absolute position of the pointsof interest by applying a global position of the augmented realitydevice to the frame coordinates and the depth coordinates of the pointsof interest; and storing the absolute position of the points of interestin the 3D data set.
 9. A method for displaying a 2-dimensional (2D)point of interest in a 3-dimensional (3D) view, comprising: providing a3D image database comprising points of interest having a 3D data setobtained by the method according to claim 1; and displaying the pointsof interest in the 3D view as augmented reality.
 10. An augmentedreality device for capturing a 3-dimensional (3D) image, comprising: anadjustable lens; an image sensor; laser 3D sensors; an absolutepositioning sensor; a display; and a processing unit configured to: in alearning phase, capture 3D images that match 2D images previously storedin a 2D image database and associate the 3D images with the 2D images inthe 2D image database; outside of the learning phase: match a 2D imagefrom the 2D image database with a 3D image captured by the image sensorbased on the 3D images associated with the 2D images in the 2D imagedatabase, the 2D image database comprising 2D images associated withpoints of interest having a 2D data set, the captured 3D imagecomprising a 2D image captured by the image sensor and associated depthinformation captured by the laser 3D sensors based on a matrix of pointscorresponding to the captured 2D image, the captured 2D image andassociated depth information being a 2D frame constructed as a vectorfield with a vector corresponding to depth associated with each point inthe 2D image, capture 3D data of the captured 3D image from the 3Dsensors and the adjustable lens when the 2D image from the 2D databasematches the captured 3D image, convert the 2D data set to a 3D data set,and display the points of interest on the display in a 3D view asaugmented reality, wherein the captured 3D image and the images in the2D data base are in spherical coordinates.
 11. A system for providing anaugmented reality, comprising: an augmented reality device according toclaim 10; and an augmented reality image server comprising: a2-dimensional (2D) database comprising images with points of interesthaving 2D data sets, and a 3-dimensional (3D) database comprising imageswith points of interest having 3D data sets.
 12. An augmented realitydevice for capturing a 3-dimensional (3D) image, comprising: anadjustable lens; an image sensor; laser 3D sensors; an absolutepositioning sensor; and a display; a 2-dimensional (2D) databasecomprising 2D images with points of interest having 2D data sets; a 3Ddatabase configured to store 3D images with points of interest having 3Ddata sets; and a processing unit configured to: in a learning phase,associate the 3D images in the 3D database with the 2D images in the 2Dimage database; outside of the learning phase: match a 2D image from the2D database with a 3D image captured by the image sensor based on the 3Dimages associated with the 2D images in the 2D image database, thecaptured 3D image comprising a 2D image captured by the image sensor andassociated depth information captured by the laser 3D sensors based on amatrix of points corresponding to the captured 2D image, the captured 2Dimage and associated depth information being a 2D frame constructed as avector field with a vector corresponding to depth associated with eachpoint in the 2D image, capture 3D data of the captured 3D image from the3D sensors and the adjustable lens when the 2D image from the 2Ddatabase matches the captured 3D image, convert the 2D data set to a 3Ddata set, and display the points of interest on the display in a 3D viewas augmented reality, wherein the captured 3D image and the images inthe 2D data base are in spherical coordinates.